Method and apparatus for providing a three-dimensional task gallery computer interface

ABSTRACT

The present invention provides a three-dimensional user interface for a computer system that allows a user to combine and store a group of windows as a task. The image of each task can be positioned within a three-dimensional environment such that the user may utilize spatial memory in order to remember where a particular task is located. In further embodiments of the invention, the three-dimensional environment includes a stage, which is used to display the task with the current focus. When a user selects a new task in the gallery, the task is moved to the stage and given focus. If a previous task was on the stage, an image of the previous task is captured. This image is then moved into the task gallery away from the stage. This process allows users to switch between multiple window configurations with a simple action.

REFERENCE TO RELATED APPLICATIONS

The present application claims priority from U.S. ProvisionalApplications having Ser. Nos. 60/128,003 and 60/127,997, both filed onApr. 6, 1999 and entitled METHOD AND APPARATUS FOR PROVIDING ATHREE-DIMENSIONAL TASK GALLERY COMPUTER INTERFACE and METHOD ANDAPPARATUS FOR PROVIDING AND ACCESSING HIDDEN TOOL SPACES, respectively.

The present application is also related to three U.S. patent applicationowned by a common assignee with the present application and filed oneven date herewith. The three applications are identified by Ser. Nos.09/541,133, 09/539,817, and Ser. No. 09/540,744 and are entitled “METHODAND APPARATUS FOR PROVIDING AND ACCESSING HIDDEN TOOLSPACES”, “METHODAND APPARATUS FOR HANDLING DISMISSED DIALOGUE BOXES”, and “METHOD ANDAPPARATUS FOR SUPPORTING TWO-DIMENSIONAL WINDOWS IN A THREE-DIMENSIONALENVIRONMENT”, respectively.

BACKGROUND OF THE INVENTION

The present invention relates to computer interfaces. In particular, thepresent invention relates to three-dimensional computer interfaces.

Many computer systems display images produced by different applicationswithin different windows on a computer monitor. Examples of operatingsystems that generate such windows include Windows 95®, Windows 98®,Windows NT®, and Windows® 2000 from Microsoft Corporation. In suchsystems, users are able to interact with multiple applications. Forexample, a user may have one window open for Word 97 from MicrosoftCorporation and a second window open for Excel from MicrosoftCorporation.

It has been observed that computer users open different windows fordifferent tasks and organize the display of their windows differentlyfor different tasks. For example, when a user performs the task ofwriting a computer program, they may have two windows open in a splitscreen format, with one window containing a program editor and the otherwindow containing the interface generated by the program. However, whenthe user is performing a task such as sending e-mails, the user may havethe mail window open so that it takes up most of the screen and ascheduling application open in a small part of the screen.

Since each task may be associated with different windows in differentlayouts, one system of the prior art has allowed the user to associatethe layout of particular windows with particular tasks. In this priorart system, such layouts were referred to as rooms, even though thelayout provided a two-dimensional view of the windows as seen on mostcurrent two-dimensional displays. The user could select one of the roomsto work with by picking the room from a grid of icons representing eachof the available rooms. In this prior art system, the rooms are placedin the grid by the system. This forces the user to scan the grid inorder to find the room that they wish to work with. Such a layout makesthe use of the room system difficult for most users.

SUMMARY OF THE INVENTION

The present invention provides a three-dimensional user interface for acomputer system that allows a user to combine and store a group ofwindows as a task. The image of each task can be positioned within athree-dimensional environment such that the user may utilize spatialmemory in order remember where a particular task is located.

In further embodiments of the invention, the three-dimensionalenvironment includes a stage, which is used to display the task with thecurrent focus. When a user selects a new task in the gallery, the taskis moved to the stage and given focus. If a previous task was on thestage, an image of the previous task is captured. This image is thenmoved into the task gallery away from the stage. In many embodiments,the image of the previous task is sent to the location in the gallerywhere the previous task resided before it was selected to move to thestage.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a plan view of a general computing environment in whichembodiments of the invention may be practiced.

FIG. 2 is a top-back perspective view of a task gallery display of anembodiment of the present invention.

FIG. 3 is a side perspective view of the task gallery of FIG. 2.

FIG. 4 is a screen image of a task gallery user interface generatedunder an embodiment of the present invention.

FIG. 5 is a diagram of a container object hierarchy under an embodimentof the invention.

FIG. 6 is a screen image of the task gallery of FIG. 4 populated bytasks and windows.

FIG. 7 is a diagram showing the relationship between mouse movement andobject movement for objects associated with different parts of the taskgallery.

FIGS. 8A-8B show selected frames from the animated movement of a task onthe right side wall of the task gallery.

FIGS. 9A-9B show selected frames from the animated movement of a task onthe left side wall of the task gallery.

FIGS. 10A-10B show selected frames from the animated movement of a taskon the floor of the task gallery.

FIGS. 11A-11B show selected frames from the animated movement of a taskon the ceiling of the task gallery.

FIGS. 12A-12I show selected frames from the animated movement of a taskas it is moved between the walls, ceiling and floor of the task gallery.

FIGS. 13A-13E show selected frames from the animated movement of taskswhen focus is shifted to a new task.

FIGS. 14A-14F show selected frames from the animated movement of thevirtual user and tasks when focus is shifted to a new task using a menuselection.

FIGS. 15A-15D show selected frames from the animated movement of avirtual user to the home viewing area.

FIG. 16 shows a movement control in a task gallery of one embodiment ofthe invention.

FIG. 17 shows a focus task from the perspective of the home viewingarea.

FIGS. 18A-18D show selected frames from the animated movement of awindow from the primary viewing location to the loose stack.

FIGS. 19A-19C show selected frames from the animated movement of awindow from the ordered stack to the loose stack.

FIGS. 20A-20C show selected frames from the animated movement of awindow from the ordered stack to the primary viewing location in placeof an existing window in the primary viewing location.

FIGS. 21A-21C show selected frames from the animated movement of awindow from the loose stack to the ordered stack.

FIGS. 22A-22C show selected frames from the animated movement of awindow from the loose stack to the primary viewing location in place ofan existing window in the primary viewing location.

FIGS. 23A-23C show selected frames from the animated movement of awindow from the primary viewing location to the ordered stack.

FIGS. 24A-24C show selected frames from the dragging of a window withinthe loose stack.

FIGS. 25A-25C show selected frames from the animated movement of awindow within the loose stack.

FIGS. 26A-26F show selected frames from the dragging and animatedmovement of a window within the ordered stack.

FIG. 27 shows a set of iconic buttons for controlling movement ofwindows in an embodiment of the present invention.

FIGS. 28A-28D show selected frames from the animated movement of awindow from the primary viewing location to the loose stack using buttonicons.

FIGS. 29A-29C show selected frames from the animated movement of awindow from the ordered stack to the loose stack using button icons.

FIGS. 30A-30C show selected frames from the animated movement of awindow from the ordered stack to the primary viewing location in placeof an existing window in the primary viewing location using buttonicons.

FIGS. 31A-31C show selected frames from the animated movement of awindow from the loose stack to the primary viewing location in place ofan existing window in the primary viewing location using button icons.

FIGS. 32A-32C show selected frames from the animated movement of awindow from the loose stack to the ordered stack using button icons.

FIGS. 33A-33C show selected frames from the animated movement of awindow from the primary viewing location to the ordered stack usingbutton icons.

FIGS. 34A-34C show selected frames from the animated movement of awindow within the loose stack using button icons.

FIGS. 35A-35C show selected frames from the animated movement of awindow within the loose stack using a second embodiment of button icons.

FIGS. 36A-36C show selected frames from the dragging of a window withinthe loose stack using button icons.

FIGS. 37A-37F show selected frames from the dragging and animatedmovement of a window within the ordered stack using button icons.

FIGS. 38A-38J show selected frames from the animated movement associatedwith adding windows to the primary viewing location.

FIGS. 39A-39C show selected frames from an animated movement associatedwith glancing at the left tool space.

FIGS. 40A-40C show selected frames from an animated movement associatedwith returning to a forward view after selecting an application from theleft tool space.

FIGS. 41A-41C show selected frames from an animated movement associatedwith glancing at the right tool space.

FIGS. 42A-42C show selected frames from an animated movement associatedwith returning to a forward view after selecting an application from theright tool space.

FIGS. 43A-43C show selected frames from an animated movement associatedwith glancing at the up tool space.

FIGS. 44A-44C show selected frames from an animated movement associatedwith glancing at the down tool space.

FIGS. 45A-45E show selected frames from an animated movement of adismissed dialog box as it moves toward the down tool space.

FIGS. 46A-46E show selected frames from an animated movement of a windowfrom one task to another.

FIGS. 47A-47B show selected frames from an animated movement of a windowboundary during resizing.

FIG. 48 is a block diagram of software and hardware elements of oneembodiment of the present invention.

FIG. 49 is a flow diagram for redirecting window display data generatedby an application.

FIG. 50 is a flow diagram of an animation loop for rendering redirectedwindow data.

FIG. 51 is a flow diagram for redirecting pointing device inputmessages.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

FIG. 1 and the related discussion are intended to provide a brief,general description of a suitable computing environment in which theinvention may be implemented. Although not required, the invention willbe described, at least in part, in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a personal computer. Generally, program modules includeroutine programs, objects, components, data structures, etc. thatperform particular tasks or implement particular abstract data types.Moreover, those skilled in the art will appreciate that the inventionmay be practiced with other computer system configurations, includinghand-held devices, multiprocessor systems, microprocessor-based orprogrammable consumer electronics, network PCs, minicomputers, mainframecomputers, and the like. The invention may also be practiced indistributed computing environments where tasks are performed by remoteprocessing devices that are linked through a communications network. Ina distributed computing environment, program modules may be located inboth local and remote memory storage devices.

With reference to FIG. 1, an exemplary system for implementing theinvention includes a general purpose computing device in the form of aconventional personal computer 20, including a processing unit (CPU) 21,a system memory 22, and a system bus 23 that couples various systemcomponents including the system memory 22 to the processing unit 21. Thesystem bus 23 may be any of several types of bus structures including amemory bus or memory controller, a peripheral bus, and a local bus usingany of a variety of bus architectures. The system memory 22 includesread only memory (ROM) 24 and random access memory (RAM) 25. A basicinput/output (BIOS) 26, containing the basic routine that helps totransfer information between elements within the personal computer 20,such as during start-up, is stored in ROM 24. The personal computer 20further includes a hard disk drive 27 for reading from and writing to ahard disk (not shown), a magnetic disk drive 28 for reading from orwriting to removable magnetic disk 29, and an optical disk drive 30 forreading from or writing to a removable optical disk 31 such as a CD ROMor other optical media. The hard disk drive 27, magnetic disk drive 28,and optical disk drive 30 are connected to the system bus 23 by a harddisk drive interface 32, magnetic disk drive interface 33, and anoptical drive interface 34, respectively. The drives and the associatedcomputer-readable media provide nonvolatile storage of computer readableinstructions, data structures, program modules and other data for thepersonal computer 20.

Although the exemplary environment described herein employs the harddisk, the removable magnetic disk 29 and the removable optical disk 31,it should be appreciated by those skilled in the art that other types ofcomputer readable media which can store data that is accessible by acomputer, such as magnetic cassettes, flash memory cards, digital videodisks, Bernoulli cartridges, random access memories (RAMs), read onlymemory (ROM), and the like, may also be used in the exemplary operatingenvironment.

A number of program modules may be stored on the hard disk, magneticdisk 29, optical disk 31, ROM 24 or RAM 25, including an operatingsystem 35, one or more application programs 36, other program modules37, and program data 38. A user may enter commands and information intothe personal computer 20 through local input devices such as a keyboard40, pointing device 42 and a microphone 43. Other input devices (notshown) may include a joystick, game pad, satellite dish, scanner, or thelike. These and other input devices are often connected to theprocessing unit 21 through a serial port interface 46 that is coupled tothe system bus 23, but may be connected by other interfaces, such as asound card, a parallel port, a game port or a universal serial bus(USB). A monitor 47 or other type of display device is also connected tothe system bus 23 via an interface, such as a video adapter 4B. Inaddition to the monitor 47, personal computers may typically includeother peripheral output devices, such as a speaker 45 and printers (notshown).

The personal computer 20 may operate in a networked environment usinglogic connections to one or more remote computers, such as a remotecomputer 49. The remote computer 49 may be another personal computer, ahand-held device, a server, a router, a network PC, a peer device orother network node, and typically includes many or all of the elementsdescribed above relative to the personal computer 20, although only amemory storage device 50 has been illustrated in FIG. 1. The logicconnections depicted in FIG. 1 include a local area network (LAN) 51 anda wide area network (WAN) 52. Such networking environments arecommonplace in offices, enterprise-wide computer network Intranets, andthe Internet.

When used in a LAN networking environment, the personal computer 20 isconnected to the local area network 51 through a network interface oradapter 53. When used in a WAN networking environment, the personalcomputer 20 typically includes a modem 54 or other means forestablishing communications over the wide area network 52, such as theInternet. The modem 54, which may be internal or external, is connectedto the system bus 23 via the serial port interface 46. In a networkenvironment, program modules depicted relative to the personal computer20, or portions thereof, may be stored in the remote memory storagedevices. It will be appreciated that the network connections shown areexemplary and other means of establishing a communications link betweenthe computers may be used. For example, a wireless communication linkmay be established between one or more portions of the network.

Under the present invention, a three-dimensional user interface isgenerated that allows a user to manipulate and use windows byassociating the windows with separate tasks. In the description below,this three-dimensional user interface is referred to alternatively as atask gallery, a data hallway, and a data mine. Generally, thethree-dimensional user interface gives the user the perception that theyare within a hallway or gallery consisting of a number of alignedhallway sections that end with a stage or display area at an end wall.

Task Gallery Layout

FIG. 2 provides a top back perspective view of a task gallery 200 of oneembodiment of the present invention with the ceiling in the gallery.removed to expose the remainder of the gallery. Task gallery 200includes rooms 202, 204, 206 and 208 that each have walls forming aportion of side walls 210 and 212, and floors that form a portion ofgallery floor 214. Room 208 also includes an end wall 216 and a stage217.

FIG. 3 provides a perspective view from the side of task gallery 200 ofFIG.2. In FIG. 3, ceiling 202 of task gallery 200 is shown connectingside walls 210, and 212. Although only four rooms, 202, 204, 206 and 208are shown in FIGS. 2 and 3, many task galleries of the present inventionare indefinitely extendable by the user. In one embodiment, the userinterface automatically generates additional rooms as the user movesobjects out of the last existing room or creates new objects thatnecessitate the creation of new rooms. In such embodiments, theinterface also removes back-end rooms if they no longer contain objects.Thus, the task gallery may consist of as few as one room.

When the user is using task gallery 200 of FIG. 3, the three-dimensionalimage provided to the user is based upon the combination of the locationof a virtual body, representing the user's body in the task gallery andthe orientation of a virtual head or camera representing the user's headin the task gallery. The user's virtual head is able to rotateindependently of the direction the virtual body is facing, so that theuser can glance up and down and to the sides as discussed further below.

FIG. 4 provides a screen image representing the view from the virtualcamera when the virtual camera is directed toward end wall 216 and thevirtual body is positioned at a location 220 in FIG. 3. Thus, in FIG. 4,end wall 216 and stage 217 are shown as being some distance from theuser and floor 214, ceiling 218, and walls 210 and 212 can be seen.

In some embodiments, each segment of the hallway is decorated so as tomake it distinct from other segments of the hallway. For example, thewalls, floor, and ceiling of a segment may be decorated with uniquetexture maps to make the hallway segment look unique. This helps toenhance the user's spatial memory of locations for storing or retrievingobjects. Segments of the hallway may also be decorated withthree-dimensional landmarks such as a virtual chair, a chandelier, orother decoration, to make the hallway segment further visually distinctand memorable.

Container Objects

In one embodiment of the invention, the user interface program thatgenerates the three-dimensional task gallery is programmed using anobject-oriented programming language. Under such an embodiment, acontainer object is defined that includes a property of containing otherobjects. The objects that are contained within a container object areknown as containables. A displayed item associated with a containableobject has its appearance and movement defined in part by the containerobject that holds the containable object.

In one embodiment, the task gallery is represented by a container objectthat contains room objects. Each room object contains two side wallobjects, a floor object and a ceiling object. Each of these containableobjects is in turn a container object that contains further objects.This hierarchy is shown in FIG. 5 where task gallery object 250 containsroom objects 251 and 252. Room object 251 contains stage object 254,left side wall object 255, right side wall object 256, end wall object257, floor object 258, and ceiling object 260. Room object 252 containsleft side wall object 270, right side wall object 272, floor object 274,and ceiling object 276. When using a task gallery, the user may add taskobjects to the wall, floor or ceiling objects of a room. For example,task objects 262, and 264 of FIG. 5 have been added to left side wallobject 270 of room object 252. When a task object is added to astructural object such as a left side wall object, the image associatedwith the task object appears bound to the image of the structureassociated with the structural object.

For example, FIG. 6 shows that a task image 300 appears near left sidewall 210 of room 208 when the task object associated with task image 300is contained within the left side wall object of the room.

The appearance of task 300 in FIG. 6 is defined in part by the fact thatthe task object representing task 300 is contained within the left sidewall object. In particular, task 300 appears on a stand 302 and has atitle bar 304 placed along its top edge. The stand helps the userdetermine the three-dimensional location of the particular task. Inaddition, task 300 does not lie flat against wall 210, but insteadextends out into the hallway of the task gallery.

In FIG. 5, task objects are shown in right side wall object 252, andceiling object 260 of room object 251 and floor object 274 of room 252.Examples of images associated with such task objects are shown in FIG. 6as right side wall task 310, ceiling task 308, and floor task 306,respectively.

In the embodiment of FIG. 6, floor task 306 appears with the top of thetask closer to the end wall than to the user. In addition, a title bar312 appears on the top edge of the task and the top edge is raisedslightly from floor 214 to provide a better view of the task to theuser.

Ceiling task 308 has its top edge closer to the user than to end wall216. The top edge of task 308 is covered by a title bar 314 and thelower edge of task 308 is suspended slightly from ceiling 218 to providea better view of the task image. All of these arrangements may becreated or changed by the user, at will, to provide arrangementsoptimized for particular uses.

Right side wall task 310 appears on a stand 318 and has a title bar 316.Task 310 does not lie flat against wall 212, but instead extends outinto the hallway of the task gallery.

Note that the specific appearances of tasks 300, 306, 308, and 310 shownin FIG. 6 are only examples of one embodiment of the present invention.The specific appearance of any one of the tasks can be changed withinthe scope of the invention. In particular, tasks on side walls 210 and212 may lie flat against the wall and may not appear with a stand. Undersome embodiments, the height of the stand changes dynamically toaccommodate the placement of the task so that the task always appears tohave a visual link with the floor area below it.

In one embodiment, structural objects such as left side wall object 255,right side wall object 256, floor object 258, and ceiling object 260 mayeach contain multiple task objects. In addition, task images associatedwith each task object may be moved along the image associated with therespective wall, ceiling or floor object that contains the task object.Moreover, a task object may be moved between container objects causingthe task image to change in response to its new container object.

Movement of Tasks Within the Gallery

The tasks and windows of FIG. 6 can be moved by the user. Table 1 belowdescribes the relationship between certain input key strokes andpointing device events to the movement of windows and tasks within atask gallery such as the task gallery of FIG. 6. The particular effectsof each input are dependent on the location of the cursor. The first twocolumns of Table 1 indicate the type of window underneath the cursor andthe task in which that window is located. The top row of Table 1indicates the type of user input provided by the input device. Althoughspecific user inputs have been listed in Table 1, those skilled in theart will recognize that other input devices can be used in place ofthose chosen and that not all affects shown within a same column for aninput instruction are necessarily required to be controlled by the sameinput instructions. In other words, simply because two events occur inthe same column in the embodiment of Table 1 does not necessarily meanthat the same events must be generated for the same input instructionsused in other embodiments.

TABLE 1 SHIFT LEFT LEFT LEFT DRAG + DRAG DRAG DRAG TASK WINDOW CLICKCLICK ALT DRAG UP DOWN LEFT RIGHT NON- N/A NO NO STEER NO NO NO NO TASKCHANGE CHANGE CAMERA CHANGE CHANGE CHANGE CHANGE NON- N/A SWITCH NO NOMOVE MOVE MOVE MOVE FOCUS TASKS CHANGE CHANGE TASK IN TASK IN TASK INTASK IN TASK GALLERY GALLERY GALLERY GALLERY FOCUS FOCUS PASS TO PASS NOPASS TO PASS TO PASS TO PASS TO TASK (CLIENT APPLIC TO CHANGE APPLICAPPLIC APPLIC APPLIC AREA) APPLIC FOCUS FOCUS NO NO NO NO NO TO NO TASK(NON- CHANGE CHANGE CHANGE CHANGE CHANGE ORDERED CHANGE CLIENT) STACKFOCUS LOOSE REPLACE ADD TO MOVE IN PULL TO PULL TO TO NO TASK STACK INPREF. LOOSE FRONT FRONT ORDERED CHANGE PREF. VIEW STACK OF OF STACK VIEWLOOSE LOOSE STACK STACK FOCUS ORDERED REPLACE ADD TO MOVE IN NO NO NO TOTASK STACK IN PREF. ORDERED CHANGE CHANGE CHANGE LOOSE PREF. VIEW STACKSTACK VIEW FOCUS NON- SWITCH SWITCH SWITCH NO NO TO NO TASK FOCUS &FOCUS FOCUS FOCUS CHANGE CHANGE ORDERED CHANGE PREF. STACK VIEW

In Table 1, there are seven different types of input instructions. Thefirst is the left click instruction in which the left button of apointing device is clicked by depressing and releasing the button. Thesecond instruction is a shift-left click, in which the shift key of thekeyboard is depressed while the left button of the pointing device isclicked. The third input instruction is a left drag plus “alt” key, inwhich the left button of the pointing device and the “alt” key of thekeyboard are depressed while the pointing device is moved. The last fourinstructions are drag up, drag down, drag left, and drag right. Theseinstructions involve depressing the left button of the pointing deviceand moving the pointing device up, down, left and right, respectively.

Those skilled in the art will recognize that other input instructionsare possible under the present invention. For instance, under oneembodiment, a secondary pointing device such as a touch pad is used toprovide input. In alternative embodiments, input instructions areindicated by using a combination of keystrokes with the arrow keys onthe keyboard.

As shown in Table 1, any task that does not have focus (i.e. any taskthat is not on stage 217) may be moved by using a traditional dragtechnique. Thus, by positioning a cursor over the desired non-focustask, and depressing the primary button of the pointing device, the usercan move the selected task by moving the pointing device. When the taskis in the desired position, the user releases the primary button to“drop” the task in its new location. As discussed further below, somewindows in the focus task can also be moved using this technique.

During a drag operation, the direction in which a task moves for a givenmovement of the pointing device is dependent upon which container objectthe task is contained within. FIG. 7 describes the relationship betweenmovement of the input pointing device and corresponding movement of atask contained by objects associated with floor 214, ceiling 218, stage217, and walls 216, 210 and 212. In FIG. 7, the arrows indicate thedirections that objects move along the respective structures and thewords by the arrows indicate the directions of movement of the pointingdevice. For walls 216, 210 and 212, movement forward and back with thepointing device results in movement of the selected task or windowupward and downward, respectively. For a task on left side wall 210,movement of the input device to the left and right causes the window tomove respectively away from and toward end wall 216. For a task on rightside wall 212, movement of the input device to the left and right causesthe task to move respectively toward and away from end wall 216. Inessence, the task currently being moved by the user will appear to staydirectly under the cursor.

For a task or window on end wall 216, stage 217, floor 214, and ceiling218, movement of the input device left and right causes the window ortask to move respectively left and right on the display. For tasks onstage 217, floor 214 or ceiling 218, movement of the input deviceforward and back causes the displayed window to move respectively towardand away from end wall 216. In one embodiment, tasks and windows arerestricted from moving freely in all three dimensions of the virtualspace but instead are restricted to two-dimensional movement on thesurface of a wall, floor or ceiling.

FIGS. 8A and 8B depict the movement of a task 350 along right side wall212. In FIG. 8A, task 350 is initially shown near the virtual user. Theuser then selects task 350 by positioning the cursor over task 350 anddepressing the primary button of the pointing device. As the user movesthe pointing device to the left, task 350 recedes toward stage 217 andis eventually dropped by the user at the location shown in FIG. 8B. Notethat because the present invention provides a three-dimensional userinterface, as task 350 is moved toward stage 217, it progressivelyappears smaller.

FIGS. 9A and 9B show the movement of a task 352 along left side wall210. FIGS. 10A and 10B show the movement of a task 354 along floor 214as task 354 is moved toward the user and away from stage 216. FIGS. 11Aand 11B show the movement of a task 356 along ceiling 218 away fromstage 216 and toward the user.

In one embodiment of the invention, tasks may be moved between the sidewall, the ceiling and the floor. Such movements are shown in FIGS. 12Athrough 12I. In FIG. 12A, a task 370 is shown on wall 212. In FIG. 12B,task 370 has been moved to the bottom of wall 212 near floor 214.Continued movement downward along wall 212 eventually causes task 370 tobe removed from wall 212 and placed onto floor 214. To avoid thepossibility that the task will flip-flop between the floor and the wallduring the transition, one embodiment of the present invention includesa hysteresis distance along the floor and the wall. Thus, the mouse mustcontinue to move a certain distance after the task meets theintersection of the floor and the wall before the task is moved to thefloor. Likewise, the window will not move from the floor back to thewall until the mouse is moved a small distance to the right of theintersection of the floor and the wall.

In an object oriented embodiment, such as the embodiment of FIG. 5, themovement of task 370 from wall 212 to floor 214 involves moving the taskobject associated with task 370 from the right side wall containerobject to the floor container object. As the task object is transferred,it loses the appearance and movement behavior dictated by the right sidewall container object and adopts the appearance and movement behaviordictated by the floor container object. Thus, stand 372, which is shownin FIGS. 12A and 12B, disappears in FIG. 12C and the orientation of task370 is changed so that task 370 leans toward stage 216 instead ofextending out into the hallway. In addition, once the task object hasbeen moved to the floor container object, left and right movement of thepointing device no longer moves the task toward and away from stage 216but instead moves the task left and right across floor 214.

In FIG. 12D, task 370 has been moved across floor 214 so that it is nextto left side wall 210. Continued movement in this direction causes thetask object associated with task 370 to be transferred from the floorcontainer object to the left side wall container object. This causes theappearance of task 370 to change as shown in FIG. 12E where task 370 isnow shown on a stand 374 and is in an upright position along wall 210.In FIG. 12F, task 370 has been moved upward along wall 210 towardceiling 218. As task 370 is moved upward, stand 374 expands so that itcontinues to connect task 370 to floor 214.

In FIG. 12G, task 370 has been moved further upward along wall 210causing the task object associated with task 370 to be removed from theleft wall container object and into the ceiling container object.Because of this, the appearance of task 370 has changed by removing thestand found in FIGS. 12E and 12F and leaning the bottom of task 370toward stage 216. Other embodiments include further changing theappearance of each task to suggest a more realistic relationship betweena task and its location in a particular room. For example, a task movedto the ceiling area might have its appearance changed so that it lookslike it is hanging from the ceiling. A task placed on the floor mightgrow legs so that it appeared to provide some semantic consistency withthe environment.

In FIG. 12H, task 370 has been moved to the right across ceiling 218toward right side wall 212. Continued movement to the right causes thetask object associated with task 370 to be removed from the ceilingcontainer object and placed into the right wall container object. Thiscauses a transformation in the appearance of task 370 as shown in FIG.12I. In particular, task 370 is again vertical in FIG. 12I and has astand 376 that extends from task 370 to floor 214.

Objects on Stage

Returning to the hierarchy of FIG. 5, it can be seen that stage object254 contains only one task object 268. In the embodiment shown in FIG.6, when a task object is placed in stage object 254, it becomes thefocus task and is associated with an image that does not have a borderaround the task nor a title bar over the task. (Although in someembodiments, the title of the task can be seen in the backdrop of thefocus task.) In addition, instead of being a single image element, atask on the stage consists of multiple window images that can each bemanipulated by the user.

The window images of the focus task have associated window objects thatare grouped into container objects within task object 268. Specifically,as shown in FIG. 5, task object 268 contains a number of other containerobjects including a loose stack object 270, an ordered stack object 272,and a primary view object 274. Each of these objects further contains acollection of window objects such as window objects 276 and 278 of loosestack object 270. One of the windows contained by primary view object274 is a focus window object 280. Focus window object 280 is associatedwith an application, which receives keyboard and appropriate pointerdevice input values as long as its associated window object isdesignated as focus window object 280.

Although multiple window objects are shown in loose stack 270, orderedstack 272 and primary view 274, these containers are not required toalways contain a window object. At different times during the practiceof the invention, each of these containers may be devoid of windowobjects.

Examples of window images associated with window objects found in afocus task, such as task 268 of FIG. 5, are shown in FIG. 6. In FIG. 6,window 320 is an image associated with a focus window object containedby a primary view object, windows 322 and 324 are associated with windowobjects contained by a loose stack object, and windows 326 and 328 areassociated with window objects contained by an ordered stack object.

In FIG. 6, window 320 appears closer to the user than loose stackwindows 322 and 324 and ordered stack windows 326 and 328. Loose stackwindows 322 and 324 each appear on stands, and ordered stack windows 326and 328 each appear on a podium 330.

Under some embodiments of the invention, various visual cues are addedto each window in order to further indicate its state. For example,windows that are not selected, and thus do not allow applicationinteraction, can be shown with a semi-transparent pane over the extentof the window. Additionally an icon in the form of a padlock can besuperimposed over the window to indicate its state.

Under one embodiment of the present invention, the user may onlyinteract directly with an application associated with awindow if thewindow is placed in the primary view associated with the stage and thewindow is given focus. Thus, in order to interact with a window within atask, the user must first place the task at the stage. Under theembodiment of Table 1, this is easily achieved by clicking on thenon-focus task that the user wishes to move to the stage. Based on thisclicking, the user interface of the present invention provides ananimated display showing the removal of the current task from the stageand its replacement by the selected task. Selected frames from such ananimation are shown in FIGS. 13A through 13E.

FIG. 13A shows an initial state of the user interface display showing acurrent task 400 having a primary viewing window 402 and two loose stackwindows 404 and 406. The initial display of FIG. 13A also includes aselected task 408, which is the task the user has “clicked” on to moveto the stage.

After the user selects task 408, the user interface generates a“snapshot” of current task 400. The snapshot of current task 400 is animage showing the appearance of task 400 from the home viewing areabefore task 408 was selected.

To produce this snap shot while maintaining the image of the galleryprovided to the user, some embodiments of the invention utilize twoimage buffers. Most often, these embodiments change the typicaloperation of two image buffers that are already present in mostthree-dimensional rendering systems. During normal operation, one ofthese buffers, known as the back buffer, is being filled with image datawhile the other buffer is being accessed by the display driver togenerate the display. The data filling the back buffer represents theappearance of the gallery from the user's next position in the gallery.When the back buffer is full, the two buffers are swapped such that thecurrent display buffer becomes the new back buffer and the current backbuffer becomes the new display buffer. The new back buffer is thencleared and filled with new image data representing the user's nextposition in the gallery.

This normal operation is changed to create the snap shot. When the userselects a new task, the camera's next position is set to the homeviewing area. The image of the task gallery is then rendered from thatposition and the rendered image data is stored in the back buffer.Without swapping the two buffers, the data in the back buffer is readout into a separate memory location that holds the snap shot. The backbuffer is then cleared and the position of the camera is reset to itsprevious position. Normal operation of the buffers is then restored.During this operation, the display buffer is accessed to produce thedisplay, so the user is unaware of the temporary change in the cameraposition.

Once generated, the snapshot is displayed on a stand as shown in FIG.13B where a task image 410 has been generated over the stage. After taskimage 410 has been generated, task image 410 begins to move away fromthe stage. In one embodiment, task image 410 moves toward its lastlocation in the task gallery before it was selected to move to stage217. In some embodiments, this last location is marked by a stand, suchas stand 412, that supports a “dimmed” or “faded” image of the task asit appeared before it was moved to the stage. In other embodiments, thelocation is not visibly marked on the display.

At the same time, task image 409 of selected task 408 begins to movetoward stage 217 while stand 411 of selected task 408 and a fadedversion of task image 409 remain in place along the right side wall.FIG. 13C shows one frame of the display during the animated movement ofboth task image 410 and selected task 408.

In some embodiments, various visual cues are placed around the border ofthe selected task to indicate that it is selected. These can include aborder, brighter background image, or additional textual cues.

As task image 410 moves from the stage, its associated task object isremoved from the stage container object and is placed in the left sidewall container object. In one embodiment, this occurs as soon as taskimage 410 moves far enough left in the animation to be considered movingalong the left side wall.

In FIG. 13D, task image 410 has returned to its previous. position inthe task gallery and selected task image 409 is positioned over stage217. When task image 409 reaches stage 217, the task object associatedwith task image 409 is removed from the side wall container it had beenin and is placed in the stage container. When task image 409 is placedin the stage container, under one embodiment, a background image that isshown behind the windows in task image 409 is expanded to fill all ofend wall 216. The windows within task image 409 are then redrawn usingcurrent data from the windows' associated applications. In FIG. 13E,this means that windows 414, 416 and 418 of selected task 408 areredrawn with the size and location of the windows determined by valuesstored for those windows when selected task 408 was last moved fromstage 217 to the task gallery.

Switching Tasks Using a Menu

In many embodiments of the invention, users may also switch betweentasks using a pop-up menu. Such a technique is shown in FIGS. 14Athrough 14F. In FIG. 14A, the user has invoked a pop-up window 420 thatprovides a “switch task” command. Although only the “switch task”command is shown in FIG. 14A, those skilled in the art will recognizethat other commands can also be present above and/or below the “switchtask” command. A secondary pop-up window 422 that provides a list oftasks available in the task gallery is shown displayed to the right ofpop-up window 420. The user may select one of the available tasks bymanipulating an input device such as the keyboard or mouse. Note that inFIG. 14A, the virtual user is in the home viewing area, which iscentered in front of stage 217 and end wall 216.

After the user has selected a task from secondary pop-up window 422, theuser interface generates an animation that gives the appearance that theuser is moving backward through the task gallery. This movementcontinues until the user is far enough back that the selected task andthe dimmed version of the former current task are fully in view. In FIG.14B, the task selected by the user is shown as selected task 424.Although not necessary to the practice of the present invention, thisautomatic movement allows the user to see an animated switch of thetasks so that the user has a better understanding of which task hasactually been selected. In one embodiment, the automatic movement of theuser can be over-ridden by the user through a user preference.

In FIG. 14C, the user interface generates a “snapshot” of the currenttask and produces task image 426 from that “snapshot”. Task image 426then begins to move toward a stand 427 at its previous location in thetask gallery. At the same time, task image 425 of selected task 424begins to move toward stage 217. FIG. 14D shows one frame during themiddle of this animated motion.

As task image 426 moves, its associated object is removed from the stagecontainer object and is placed in the left side wall container object.

When task image 426 has returned to its original location and selectedtask 424 has moved to stage 217, as shown in FIG. 14D, the objectassociated with selected task 424 is removed from the right side wallcontainer object and is placed into the stage container object. Thedisplay then regenerates each window in selected task 424 above stage217. In some embodiments, the virtual user is then returned to the homeviewing area.

Virtual User Movement

In the embodiment of the present invention associated with the inputcontrols of Table 1, the user may move through the task gallery using apointing device to indicate the direction and. duration of eachmovement. Alternatively or in addition to the direct movement, the usermay initiate movements to fixed positions within the task gallery. Tofacilitate such movement, the task gallery is divided into rooms withone or more user positions within each room. By using a single keystroke, the user may advance forward one room or backward one room. Inaddition, by using a dedicated key or dedicated combination of keys, theuser may move directly from any location within the task gallery to thehome viewing area in front of stage 217.

These embodiments of the invention provide a high level of control wherea single click on an appropriate navigational control (button), causesthe virtual user to move swiftly but smoothly from their currentlocation to a new desired location. In other words, a discrete actionresults in transportation of the virtual user to commonly used anduseful locations. This avoids problems of hand-eye coordination and theneed for well-developed spatialization skills.

FIGS. 15A through 15D show an animated motion from a remote location inthe task gallery to the home viewing area. Specifically, in FIG. 15A,the user is located in the second room of the task gallery. The userthen initiates the command to move to the home viewing area. FIGS. 15Band 15C show selected frames in the animated movement towards stage 217and FIG. 15D shows the view from the home viewing area.

FIG. 16 shows another embodiment of the present invention in which a setof movement controls 428 are displayed in the lower left corner of thethree-dimensional environment. The movement controls include a forwardarrow control 429, a backward arrow control 431, a home viewing areacontrol 433, an overview control 435, an up glance control 437, a downglance control 439, a left glance control 441, a right glance control443 and a human FIG. 445. Although the appearance of the buttons andicons in FIG. 16 are found in several embodiments of the invention,different designs for the appearance of the buttons and icons can beused depending on the intended experience level of the user.

By placing the cursor over a control and depressing a button on themouse or keyboard, a user can select the control and cause the user tomove through the environment or change the direction of their view ofthe environment. For instance, selecting forward arrow control 429causes the user to move forward one room in the environment andselecting backward arrow control 431 causes the user to move backwardone room. Selecting home viewing area control 433 causes the user tomove to the home viewing area. Selecting overview control 435 causes theuser to move to the back of the task gallery so that the entire taskgallery is visible. Selecting glancing controls 437, 439, 441, and 443is discussed below in connection with glances.

Under one embodiment of the present invention, movement controls 428 arealways present on the screen. In other embodiments, movement controls428 are only displayed when the user requests that they be displayed.For example, a touch-sensitive input device can be used to fade in orfade out the human figure. When the user touches the input device, thefigure appears, and when the user lets go it vanishes. In still otherembodiments, the human figure is always present on the screen but themovement controls only appear when the user places the cursor over thefigure. In further embodiments of the invention, pausing the cursor overone of the controls or the human figure generates a tool-tip thatdescribes the function of the control.

Further embodiments of the invention rely on input devices that areoptimized for the task of navigation. These include dedicated keys onthe keyboard, touch-sensitive pads for direction control, and/or smallspring-loaded levers with sensors to control the primary locomotioninteractions.

The Focus Task

FIG. 17 shows a screen display produced when the user is in the homeviewing area in front of stage 217. In FIG. 17, a task 430 is shown thatcontains a focus window 432 in the primary viewing area, windows 434 and436 in a loose stack area and windows 438 and 440 in an ordered stackarea. Although the screen display of FIG. 17 is described in connectionwith a virtual user placed in a three-dimensional task gallery, theinventive aspects of the screen display discussed below are not limitedto such an environment. As such, the inventive aspects of the screendisplay of FIG. 17 discussed further below may be practiced in anenvironment that does not include a three-dimensional task gallery suchas a simple three-dimensional desktop.

Moving Windows from the Primary View to the Loose Stack

Within the current or focus task, the user may move windows between theprimary viewing area, the loose stack, and the ordered stack. FIGS. 18Athrough 18D show selected frames representing the movement of a windowfrom the primary viewing area to the loose stack. In FIG. 18A, the userhas placed the cursor over window 442, which is located in the primaryviewing area. Note that window 442 has focus in FIG. 18A, and as such,most keyboard and pointing device inputs are provided directly to theapplication corresponding to the focus window. In order to overcome thisdefault, a combination of keystrokes may be used as the command to movethe window from the primary viewing area to the loose stack. Forexample, in the embodiment associated with Table 1, the user performs adrag up on the window in the primary viewing area while depressing the“alt” key in order to move it to the loose stack. Alternatively, thecommand for moving a window to the loose stack from the primary viewingarea can require that the cursor be positioned in a non-client area(also known as a window decoration area) in order for the command inputto be directed away from the application and to the user interface.

Upon receiving the input corresponding to the user's desire to movewindow 442 to the loose stack, the user interface begins to push window442 back toward a loose stack area 450 as shown in FIG. 18B. When window442 reaches loose stack area 450, as shown in FIG. 18C, the windowobject associated with window 442 is removed from the primary viewingcontainer and placed into the loose stack container. Since windows inthe loose stack have stands that connect the windows to the floor, astand is then drawn below window 442 as shown in FIG. 18D.

Moving Windows from the Ordered Stack to Loose Stack

FIGS. 19A through 19C show selected frames of an animation produced byan embodiment of the present invention to show the movement of a window454 from an ordered stack 456 to a loose stack 458. In FIG. 19A, theuser has positioned a cursor 460 over window 454. With the cursorpositioned over window 454, the user provides input corresponding to adesire to move window 454 to loose stack 458. In the embodiment of Table1 this input is a drag to the right. In other embodiments, any draggingoperation from the ordered stack toward the loose stack will beinterpreted as a command to move the selected window from the orderedstack to the loose stack.

When the user interface receives the drag right input, it generates ananimated movement of the selected window 454 that shows window 454moving up from the ordered stack 456 toward loose stack 458. Inaddition, the animation shows the rotation of window 454 so that thewindow's orientation matches the orientation of the loose stack windows.FIG. 19B shows a frame of this animated movement.

In FIG. 19C, window 454 is positioned within loose stack 458. At thispoint, the object associated with window 454 has been removed from theordered stack container and has been placed in the loose stackcontainer. As such, window 454 is drawn in the display with a stand 460extending from the bottom of window 454 to the floor. In addition, ifthe window removed from the ordered stack was not the front window, ananimation is invoked to re-position the windows in the ordered stack sothat they are a fixed distance apart from each other.

Movement of a Window from the Ordered Stack to the Primary Viewing Area

FIGS. 20A through 20C show separate frames of an animation created bythe present user interface when the user wishes to replace the window inthe primary viewing area with a window on the ordered stack. In FIG.20A, the user has positioned a cursor 462 over a window 464 in anordered stack 466. With the cursor in this position, the user indicatestheir desire to replace window 468 of the primary viewing area withwindow 464. In the embodiment of Table 1, the user indicates theirdesire for this change by clicking a primary button of a pointing devicesuch as a mouse or a track ball.

Upon receiving the “click” input, the user interface simultaneouslymoves window 464 up toward the primary viewing area and pushes window468 back toward either loose stack 470 or ordered stack 466. In oneembodiment, window 468 is pushed back toward the stack that the windowwas in before it was moved to the primary viewing area. When window 464reaches the primary viewing area and window 468 reaches loose stack 470,the object's associated with these windows are moved into theappropriate container objects. For example., window 464 is moved fromthe ordered stack container object into the primary viewing areacontainer object. In addition, window 464 is identified as the focuswindow.

Lastly, if the window removed from the ordered stack was not the frontwindow, an animation is invoked to re-position the windows in theordered stack so that they are a fixed distance apart from each other.

Moving a Window from the Loose Stack to the Ordered Stack

FIGS. 21A through 21C show frames from an animation generated when theuser indicates that they want to move a window 472 from a loose stack474 to an ordered stack 476. In one embodiment, the user indicates thatthey wish to move a window from the loose stack to the ordered stack byplacing a cursor over the window and performing a drag left. In otherembodiments, any dragging operation from the loose stack to the vicinityof the ordered stack will be interpreted as a command to move theselected window from the loose stack to the ordered stack.

After receiving the drag left input, the user interface generates ananimation in which window 472 is brought forward toward ordered stack476 and is rotated so that it is aligned with the other windows inordered stack 476. FIG. 21B shows one frame of that animated movement.Before moving window 472, stand 478 of FIG. 21A is removed from thebottom of window 472. When window 472 reaches ordered stack 476, theobject associated with window 472 is removed from the loose stackcontainer object and is placed in the ordered stack container object.

Moving a Window from the Loose Stack to the Primary Viewing Area

FIGS. 22A through 22C show selected frames from an animation generatedby the present interface when the user wishes to replace a window 484 inthe primary viewing area with a window 480 from a loose stack 482. Inthe embodiment of Table 1, the user initiates this movement by clickingon window 480. Based on this input, the user interface generates ananimation in which window 480 is brought forward from loose stack 482 tothe primary viewing area and window 484, which is in the primary viewingarea, is moved back to either the loose stack or the ordered stackdepending on where it was before being placed in the primary viewingarea. For the purposes of FIGS. 22A through 22C, window 484 was in loosestack 482 before being moved to the primary viewing area.

During the animation, the object associated with window 480 is removedfrom the loose stack container object. This causes stand 486 todisappear so that window 480 appears unsupported. At the same time, theobject associated with window 484 is removed from the primary viewingcontainer and placed into the loose stack container. When the objectassociated with window 484 is placed in the loose stack containerobject, a stand appears below window 484.

Moving Windows from the Primary Viewing Area to the Ordered Stack

FIGS. 23A through 23C show selected frames from an animation created byan embodiment of the present user interface when the user indicates thatthey wish to move a window 490 from the primary viewing area to orderedstack 492. In one embodiment, the user indicates that they want to movewindow 490 to ordered stack 492 by performing a drag left while the“alt” key is depressed and the cursor is positioned over window 490.

Under this embodiment, when the user interface receives a drag left and“alt” key input while the cursor is positioned over window 490, the userinterface initiates an animation in which window 490 is pushed backwardand rotated slightly to align itself with ordered stack 492 as shown inFIG. 23B. When window 490 reaches ordered stack 492, the objectassociated with window 490 is placed in the ordered stack containerobject and is removed from the primary viewing area object. The endresult of this animation is shown in the frame of FIG. 23C.

Moving Objects within the Loose Stack

Windows within the loose stack inherit movement properties from theloose stack container object that allow the user to reposition thewindows freely within the loose stack. In one embodiment, there are twotypes of possible movement for a window within the loose stack. First,the window may be moved laterally or vertically within the loose stackas shown in FIGS. 24A through 24C where window 500 in loose stack 502 ismoved by the user from an initial position shown in FIG. 24A to a secondposition shown in FIG. 24B and finally to a third position as shown inFIG. 24C. In the movement from the initial position of FIG. 24A to thesecond position of FIG. 24B, the user mostly moves the window laterallyto the right. In the second motion, from the second position of FIG. 24Bto the third position of 24C, the user moves window 500 downward and tothe left. Note that as window 500 is moved, a stand 504 located belowwindow 500 is adjusted so that it remains below window 500 and has theappropriate size to connect window 500 to the floor.

In the embodiment of Table 1, the movement shown in FIGS. 24A through24C is accomplished by the user by placing the cursor over window 500and performing a drag operation with the “alt” key depressed.

Windows within the loose stack may also be moved forward and backwardwithin the loose stack. FIGS. 25A through 25C show the movement of aloose stack window 506 first to the front of the loose stack and then tothe back of the loose stack. Thus, in FIG. 25A, window 506 is showninitially positioned between windows 508 and 510 of loose stack 512. InFIG. 25B, window 506 has been brought to the front of loose stack 512and is now in front of window 508. In FIG. 25C, window 506 has beenplaced at the back of the loose stack behind both window 510 and window508.

In the embodiment of Table 1, the user indicates that they wish to pulla loose stack window to the front of the loose stack by performing adrag down. To push a window back in the loose stack, the user performs adrag up operation.

Movement within the Ordered Stack

Under the present invention, a user can also reorganize the order ofwindows in an ordered stack. Although the user can change the order ofthe windows, the precise locations of the windows are determined by theuser interface.

For example, in FIGS. 26A through 26F, the user reorders an ordered.stack 514 that contains windows 516, 518 and 520. In the initial displayshown in FIG. 26A, window 518 is shown between windows 516 and 520 inordered stack 514. By selecting window 516, the user is able to dragwindow 516 upward as shown in FIG. 26B and forward of window 520 asshown in FIG. 26C.

Since the user interface automatically repositions windows in theordered stack, the user may release window 518 outside of the orderedstack as shown in FIG. 26D, where cursor 522 has been moved from window518 after the user has released the primary button of the pointingdevice.

When the user releases window 518, windows 518 and 520 begin to move.Specifically, window 520 moves backward in ordered stack 514 to assumethe position that window 518 originally had in ordered stack 514. At thesame time, window 518 moves downward and back toward ordered stack 514.When the movement is complete, window 518 occupies the space that window520 occupied initially in FIG. 26A. Its final resting position is shownin FIG. 26F. Thus, the user is able to reorganize ordered stack 514without having to expend unnecessary energy in realigning the windowswithin ordered stack 514.

Movement Using Icon Control

Under an alternative embodiment, windows in the primary task may also bemoved by using a set of selectable button icons such as buttons 524 ofFIG. 27. In one embodiment, buttons 524 appear on top of a window when acursor crosses into the window. The buttons persist above the window sothat the user may reposition the cursor over one of the buttons. Byclicking on one of the buttons, or by dragging one of the buttons, theuser is then able to move the selected window in any of the mannersdescribed above.

For example, a user can move a window in the primary viewing area to theloose stack by clicking loose stack button 526 of buttons 524. FIGS. 28Athrough 28D show selected frames of an animation created by anembodiment of the present invention showing the movement of a window 550from a primary viewing area to a loose stack 552 using a set of buttons554. In FIG. 28A, button icons 554 have been generated by the userinterface above window 550 based on the location of a cursor 556 withinthe window 550. The user has moved the cursor 556 over to “loose-stack”button 554 and has clicked on that button. In FIG. 28B, window 550 hasbeen pushed backward toward loose stack 552. In FIG. 28C, window 550 hasreached loose stack 552 and in FIG. 28D a stand has appeared belowwindow 550.

Loose stack button 526 may also be used to move a window from an orderedstack to the loose stack as shown in FIGS. 29A through 29C. In FIG. 29A,the user has caused button icons 560 to appear above window 562 inordered stack 564 by placing the cursor in window 562. The user has thenpositioned the cursor over loose stack button 566 of button icons 560.By clicking on loose button 566, the user initiates an animation inwhich window 562 moves to loose stack 568. FIG. 29B shows one frameduring that animated motion and FIG. 29C shows window 562 in its finalposition in loose stack 568.

Button icons 524 of FIG. 27 also include a primary view button 528,which is used to replace the current window in the primary view with aselected window. FIGS. 30A through 30C show the use of a primary viewbutton 570 to replace a window 572 in the primary view with a window 574from an ordered stack 576. In FIG. 30A, button icons 578 have beengenerated when the user moved the cursor over window 574. The user hasthen moved the cursor over the primary view button 570. When the userclicks on primary view button 570, window 572 in the primary viewingarea begins to move back toward loose stack 580 while window 574 movesforward. At the end of the movement, window 572 is located in loosestack 580 and window 574 is located in the primary viewing area.

Primary view button 528 of FIG. 27 can also be used to move a windowfrom a loose stack to the primary view. FIGS. 31A through 31C showselected frames of an animation depicting such an event. In particular,FIG. 31A shows a cursor 590 placed over a primary view button 592 ofbutton icons 594. Button icons 594 were generated by the user interfacein response to cursor 590 being positioned over loose stack window 596.

When the user clicks on primary view button 592, window 596 movesforward and current window 598 moves back toward the loose stack. FIG.31B shows a frame during a portion of this movement. FIG. 31C shows thefinal orientation of windows 596 and 598.

A user can add additional windows to the primary viewing area withoutremoving existing windows from the primary viewing area by usingadd-to-selection button 536 of FIG. 27. When the user selects thisbutton for a window in the loose stack or the ordered stack, the windowmoves to the primary viewing area and the existing windows in theprimary viewing area are moved to accommodate the new window. In oneembodiment, the windows in the primary viewing area are positioned sothat they appear to be the same size as discussed further below.

Button icons 524 of FIG. 27 also include an ordered stack button 530that can be used to move windows from the primary viewing area to theordered stack and from the loose stack to the ordered stack. FIG. 32Athrough 32B show selected frames of movement of a window 600 from theloose stack to the ordered stack. The movement of window 600 isinitiated by the user clicking on ordered stack button 602 of buttonicons 604. FIGS. 33A through 33C show the movement of a window 606 fromthe primary viewing area to ordered stack 608 when the user clicks onordered stack button 610 of button icons 612.

Button icons 524 of FIG. 27 also include a push back/pull forward button532. As shown in FIGS. 34A through 34C, the user can use button 532 topush a window in a loose stack back or pull the window forward in theloose stack. In FIG. 34A, the user has selected push back/pull forwardbutton 616 of button icons 618, which was displayed when the user placedthe cursor over loose stack window 620. While depressing the primarybutton on a pointing device, the user can pull loose stack window 620forward in the loose stack by moving the pointing device backward. Theresult of such an operation is shown in FIG. 34B, where loose stackwindow 620 is shown at the front of the loose stack. The user may alsopush loose stack window 620 to the back of the loose stack by moving thepointing device forward. The result of this operation is shown in FIG.34C.

In an alternative embodiment, push back/pull forward button 532 of FIG.27 is divided into an upper selectable arrow 531 and a lower selectablearrow 533. As shown in FIG. 35A, when a window 621 is located in themiddle of a loose stack both the upper selectable arrow and the lowerselectable arrow are shown in push back/pull forward button 617 ofbutton icons 619. By positioning the cursor, the user can select eitherthe upper arrow or the lower arrow. If the user selects the upper arrow,window 621 is pushed to the back of the loose stack as shown in FIG.35C. If the user selects the lower arrow, window 621 is pulled to thefront of the stack as shown in FIG. 35B. In one embodiment, the upperarrow and the lower arrow are rendered in three-dimensional perspectivesuch that the upper arrow appears smaller than the lower arrow. Thishelps to indicate to the user that the upper arrow will push windows tothe back and that the lower arrow will pull windows to the front.

When window 621 is at the front of the stack, the lower arrow is removedfrom button 617 as shown in FIG. 35B. Similarly, when window 621 is atthe back of the loose stack, the upper arrow is removed from button 617as shown in FIG. 35C.

Button icons 524 of FIG. 27 also include a move button 534, which theuser may use to relocate a window within the loose stack or the orderedstack. FIGS. 36A through 36C show movement of a loose stack window 624using a location button 626 of button icons 628. In FIG. 36A, the userhas selected location button 626 from button icons 628. While depressinga primary button on a pointing device, the user is able to move window624 vertically and laterally within the loose stack. As shown in FIG.36B, the user has moved window 624 laterally within the loose stack. Asshown in FIG. 36C, the user has moved window 624 down and to the leftwithin the loose stack.

The move button may also be used to provide arbitrary movement in depthwhile dragging the button. In one specific embodiment, holding the shiftkey while dragging causes the window to move away from the user andholding the control key while dragging causes the window to move towardthe user.

A move button may also be used to reorder windows within an orderedstack as shown in FIGS. 37A through 37F. In FIG. 37A, the user hasselected move button 630 of button icons 632. While depressing a primarybutton on a pointing device, the user can move window 634 as shown inFIGS. 37B and 37C by moving the pointing device. In FIG. 37D, the userhas released the primary button of the pointing device and moved thecursor away from button 630. This in turn has caused button icons 632 todisappear in FIG. 37D.

In FIG. 37E, the user interface automatically moves windows 636 and 634within the ordered stack. As shown in FIG. 37E, this involves movingwindow 636 back in the ordered stack and moving window 634 down towardthe ordered stack. FIG. 37F shows the result of the reordering done bythe user and the automatic positioning done by the user interface.

A user may also close a window using a close button 537 of icons 524.When a user clicks on close button 537, the window associated with thebutton icons disappears from the screen along with the button icons.

The order of the button icons shown in FIG. 27 represents only a singlepossible embodiment. Other orders for these buttons are within the scopeof the invention. In addition, the buttons may be arranged in otherpossible layouts within the scope of the present invention. For example,the buttons may be arranged in an arc around one of the corners of thewindow. This aids the user in consistently and quickly acquiring thebuttons for purposes of interaction.

Various embodiments of the present invention also use a variety ofstrategies for attaching the button icons to a window. In oneembodiment, the button row moves in all three dimensions with the windowsuch that when the window moves away from the user, the button rowappears to get smaller. In some embodiments, the row of buttons tiltswith the window as the window tilts. In further embodiments, the buttonrow tilts as the window tilts but during the tilt operation the buttonsare simultaneously resized and rearranged such that each button remainsa constant size (in pixels) on the screen and the spacing between thebuttons remains constant in pixels.

When an embodiment is used where the row of buttons does not tilt ormove forward and back with the window, various visual cues can be usedto suggest the association between the row of buttons and the selectedwindow. For example, semi-transparent geometric objects can stretchbetween the boundary of the row of buttons and the top edge of theselected window. Alternatively, lines may be drawn between each buttonand an associated location on the selected window. In furtherembodiments, various combinations of lines and planar objects are usedtogether to further the visual correspondence.

Multiple Windows in the Primary Viewing Area

Under an embodiment of the present invention, multiple windows can beplaced in the primary viewing area. FIGS. 38A through 38J depictselected frames showing the placement of multiple windows in the primaryviewing area. In FIG. 38A, the user has positioned a cursor 650 over aloose stack window 652. The user then indicates that they wish to addwindow 652 to the primary viewing area. In the embodiment of Table 1,this is accomplished by depressing the shift key on the keyboard whileclicking the primary button of the pointing device. In the pop-up menuembodiment of FIG. 27, this is accomplished by selecting add windowbutton 536.

In response to this input, the user interface of the present inventionpushes current focus window 654 back in the display while bringing loosestack window 652 forward in the display. A frame from this motion isshown in FIG. 38B. As loose stack window 652 is moved into the primaryviewing area, the object associated with window 652 is removed from theloose stack container object and is placed into the primary viewcontainer object. In addition, window 652 is designated as the focuswindow in the primary viewing area.

When window 652 reaches the primary viewing area, it is the samedistance from the user as window 654 with which it shares the primaryviewing area. Thus, the user does not have to manipulate the shape orlocation of either window in order to view both windows in the primaryviewing area. The result of moving window 652 into the primary viewingarea is shown in FIG. 38C. In other embodiments, the windows are placedat different distances from the user so that the windows appear the samesize to the user and so that the windows do not obscure each other. Instill, other embodiments, the windows are scaled so that they appear thesame size in the primary viewing area. In the context of thisapplication, such scaling can be considered a way of positioning thewindows.

More than two windows may be added to the primary view. In FIG. 38D theuser positions cursor 650 over an ordered stack window 656 and indicatesthat they wish to add that window to the preferred viewing area. Usingthe embodiment of Table 1, this involves pressing the shift key whileclicking the primary button of the pointing device. In the embodiment ofFIG. 27, this involves selecting the add-to-selection button 536 ofbutton icons 524. In response to the user input, the user interfacepushes windows 652 and 654 back in the display while bringing windows656 forward and to the right. A frame from this motion is shown in FIG.37E. In FIG. 38F, it can be seen that each of the windows 652, 654, and656 in the primary viewing area are of generally the same size andshape. The repositioning of the windows is done automatically by theuser interface of the present invention so that the user does not haveto manipulate these features of the windows in order to view all of thewindows in the primary viewing area. In one embodiment, window 656 isgiven focus as it is moved into the primary viewing area.

A fourth window may be added to the primary viewing area by selecting anadditional window to add to the primary viewing area as shown in FIG.38G. In FIG. 38G, the user has selected a window 660 to add to theprimary viewing area. In FIGS. 38H and 38I, window 660 is moved forwardtoward a preferred viewing area defined by windows 652, 654 and 656. InFIG. 38J window 660 reaches its final position within the preferredviewing area and is designated as the focus window.

The present invention is not limited to any particular number of windowsthat may be added to the primary viewing area. For example, in oneembodiment ten windows may be placed in the primary viewing area.

Movement of the Loose Stack and Ordered Stack

In some embodiments, the locations of the ordered stack and/or the loosestack are changed dynamically as windows are moved into and out of theprimary viewing area. This movement is designed to keep at least a partof both the loose stack and the ordered stack in view when windows areplaced in the primary viewing area.

Glances

Embodiments of the present invention utilize a glancing technique toallow the user to look ephemerally to their left and right and up anddown. For example, under one embodiment, if the user clicks on leftglance control 441 of FIG. 16, an animation is started that rotates thecamera to the left. The user is then able to see the area to the left ofthe virtual user. When the camera has been rotated ninety-degrees, theimage is held for one second and then a second animation is generated tosimulate the rotation of the camera back to the forward position.Similar glancing animations can be invoked to view the spaces to theright, above and below the virtual user by clicking on glancing controls443, 437 and 439 respectively. Any one of these glances can be held byclicking and holding on the respective control. When the control isreleased, the rotation animation toward the forward view is invoked.

In some embodiments, glancing can be used to expose tool spaces thattravel with the virtual user in the task gallery. The techniques forgenerating such tool spaces and for implementing glances to the toolspaces are discussed in detail in a U.S. patent application Ser. No.09/541,133 entitled “A METHOD AND APPARATUS FOR PROVIDING AND ACCESSINGHIDDEN TOOL SPACES” filed on even date herewith, assigned to a commonassignee.

In summary, a tool space is a container object that contains anddisplays images of other objects. The tool space container object isdifferent from other container objects described above in that the toolspace container object travels with the virtual user and can be seen byusing a glancing technique or by activating a tool space control. In aglancing technique, the camera associated with the virtual user isrotated while the virtual user's body remains in a fixed position. Ifthe virtual user's body is rotated toward the tool space, the tool spacerotates with the user such that the user does not see the tool space. Toinvoke a glance, the user utilizes a glancing gesture, which can involvea combination of keystrokes, a combination of keystrokes and pointingdevice inputs, just primary pointing device inputs, or the use of asecondary pointing device such as a touch pad. In some embodiments,glancing is invoked using movement controls 428 of FIG. 16.Specifically, glancing controls 437, 439, 441, and 443 are used toinvoke glances up, down, left, and right, respectively.

In other embodiments, the user displays a tool space without performinga glancing gesture. For example, in one embodiment, the user can displaya tool space by selecting the hands of the displayed figure 445 in FIG.16. In one such embodiment, the system displays an animation in whichthe tool space rotates into the user's current view. In such cases, whenthe user invokes a glance to the left or right they see the left andright side walls but do not see a tool space. The tool space can bedismissed by clicking on the tool space control again or by selecting anobject in the tool space. When the tool space is dismissed, an animationis displayed in which the tool space appears to return to the place itoriginally came from.

Glances to the Three-Dimensional Start Palette

FIGS. 39A through 39C show selected frames from an animated glancetoward a left tool space. In FIG. 39A, the virtual user is positioned infront of stage 217 at the home viewing location. Upon receiving theglance gesture, the user interface rotates the view to the left suchthat windows 680 and 682 rotate to the right in the display. As the viewrotates left, the left tool space comes into view. In the embodiment ofFIGS. 39A, 39B, and 39C the left tool space is depicted as a palette684. In FIG. 39C, the rotation is complete so that all of palette 684can be seen. In the embodiment of FIG. 39C, the user's hand is shownholding palette 684 to give the user a sense of depth perception as tothe location of palette 684, and to indicate the size of palette 684.

Palette 684 of FIG. 39C contains a number of three-dimensional objectssuch as objects 688 and 670. Objects 670 and 688 may be moved by placinga cursor over the object and using a dragging technique.

In one embodiment, palette 684 is a data mountain as described in aco-pending U.S. patent application having Ser. No. 09/152,491, filed onSep. 14, 1998, and entitled METHODS, APPARATUS AND DATA STRUCTURES FORPROVIDING A USER INTERFACE, WHICH EXPLOITS SPATIAL MEMORY INTHREE-DIMENSIONS, TO OBJECTS.” In such an embodiment, objects, such asobjects 670 and 688 are prevented from being moved such that one objectobscures another object. In particular, if an object begins tosubstantially cover another object, the other object moves to the sideso that it remains in view.

Selecting an Application from the Three-Dimensional Start Palette

In one embodiment, the objects on a start palette such as palette 684represent applications that can run on the same computer that isgenerating the three-dimensional computer interface of the presentinvention. By clicking on an object such as object 690 in FIG. 40A, auser of the present invention can cause the application to beginexecuting. If the application then opens a window, the present inventionwill redirect the window that is drawn by the application so that thewindow appears in the primary viewing area of the current task. The userinterface of the present invention then dismisses the tool space eitherby rotating the tool space out of the user's forward view (non-glancingtool space embodiments) or by rotating the user's view from the sideglance (glancing tool space embodiments) back to the primary task sothat the user may see the newly opened window.

FIG. 40B shows the beginning of a rotation back to the primary task froma side glance and FIG. 40C shows a return to the full view of theprimary task showing newly opened window 692 which is associated withapplication 690 of FIG. 40A. Because palette 684 can include a number ofobjects representing applications, it serves the function of currenttwo-dimensional Start Menus and favorites. Thus, palette 684 can beviewed as a three-dimensional Start Menu.

In some embodiments, the user can launch multiple applications during asingle viewing of the start palette. In one specific embodiment, theuser holds the shift key while selecting individual items. Instead oflaunching the selected items, the system changes the appearance of theicons to mark the icons as having been selected. When a user clicks onan already marked item, the tool space is dismissed and all of theselected applications are launched.

Although the left tool space has been described in connection withpalette 684, those skilled in the art will recognize that the tool spacecan take any shape.

Glancing to the Right Tool Space

In one embodiment, the task gallery also includes a right tool space,which the user can rotate to using a glancing gesture to the right. Thiscauses the rotation of the display as shown in FIGS. 41A, 41B and 41C.FIG. 41A shows an initial view of the current task on stage 216. FIG.41B shows a rotation to the right exposing a portion of a right toolspace 700. FIG. 41C shows the complete rotation to the right tool space700.

In the embodiment of FIG. 41C, right tool space 700 is a single window,which is generated by a file manager program such as Windows Explorerfrom Microsoft Corporation. In FIG. 41C, a hand 702 is shown holding awindow of tool space 700. Hand 702 gives the user some perspective onthe size and position of tool space 700 relative to their viewpoint. Asthose skilled in the art will recognize, tool space 700 can take on manydifferent appearances and the appearance shown in FIG. 41C is only oneexample.

In an embodiment in which the right tool space contains a file managersuch as the menu provided by Microsoft Windows Explorer, the user mayinvoke an application or open a document simply by selecting theapplication or document's entry in the file list.

As shown in FIGS. 42A through 42C, if the user selects an application ordocument in the file list, the application will be started and if theapplication has an associated window, the window will be put in theprimary viewing area of the current task. For example, in FIG. 42A wherethe user selects an entry 704 from file manager 706, the applicationassociated with that entry is started. The user interface then rotatesthe view back to the current task as shown in FIGS. 42B and 42C toexpose a window 708, which was created by the selected application andredirected to the primary viewing area by the user interface.

Glancing at the Up Tool Space

Embodiments of the present invention can also include an up tool space,which may be accessed by performing an upward glancing gesture. FIGS.43A through 43C depict frames from an animation that is generated when auser performs an upward glancing gesture. Specifically, FIG. 43A showsan initial view of a current task on stage 216. Upon receiving theupward glancing gesture, the user interface rotates the view upwardcausing the windows of the current task to move downward. As shown inFIG. 43B, this gradually exposes the up tool space until the entire uptool space 712 becomes visible as shown in FIG. 43C.

Glancing at the Down Too Space

Some embodiments of the invention also include a down tool space, whichmay be accessed using a downward glancing gesture. FIGS. 44A through 44Cshow frames of an animated rotation downward to expose the down toolspace. In particular, FIG. 44A shows an initial view of a current taskon stage 216. FIG. 44B shows a frame from the middle of the downwardrotation to the down tool space showing a portion of down tool space714. FIG. 44C shows the result of the full rotation to the down toolspace 714.

Down tool space 714 of FIG. 44C includes an image of shoes 716 and 718meant to depict the virtual shoes of the user in the task gallery. Inaddition, down tool space 714 includes two past dialog boxes 720 and722. Although shoes 716 and 718 and dialog boxes 720 and 722 are shownin down tool space 714, those skilled in the art will recognize thatnone of these items necessarily need to appear in down tool space 714and that other items may be added to down tool space 714 in place of orin addition to the items shown in FIG. 44C.

Movement of Past Dialog Boxes to the Down Tool Space

The present inventors have recognized that in current operating systems,users may dismiss dialog boxes that contain valuable information beforethey really know what the boxes contain. Unfortunately, once the dialogbox is dismissed, the user is not able to recover the text of the box.

To overcome this problem, an embodiment of the present invention storespast dialog boxes in the down tool space. Thus, past dialog boxes 720and 722 in FIG. 44C are examples of dialog boxes that have beendismissed by the user.

In further embodiments of the invention, the user interface generates ananimated motion of the dismissed dialog box toward the down tool spaceto indicate to the user that the dialog box has been moved to this toolspace. FIGS. 45A through 45E provide selected frames of this animatedmotion.

In FIG. 45A, a dialog box 730 is shown in the display. After the userdismisses the dialog box either by hitting enter or by selecting adisplay button within the dialog box, the user interface creates ananimation in which dialog box 730 slowly drifts to the bottom of thescreen as shown in FIGS. 45B, 45C, and 45D. Eventually, the dialog boxdrifts completely out of view as shown in FIG. 45E. If the user wishesto view the dialog box again, they execute a downward glancing gestureto access the down tool space as described above for FIGS. 44A through44C.

Under some embodiments, the number or age of the dismissed dialog boxesdisplayed in the down tool space is controlled by the system. Thus,under one embodiment, dialogue boxes are removed from the down toolspace after some period of time. In other embodiments, the oldestdialogue box is removed when a new dialogue box enters the down toolspace.

Although the dismissed dialogue boxes are shown drifting to a down toolspace, in other embodiments, the dismissed dialogue boxes move to otheroff-screen tool spaces. In addition, although the placement of dismisseddialogue boxes in a tool space is described in the context of athree-dimensional task gallery, this aspect of the invention may bepracticed outside of the task gallery environment.

Movement of a Window from One Task to Another

Under an embodiment of the invention, the user may move a window fromone task to another. In one embodiment, the user initiates such a moveby invoking a menu using a secondary button on a pointing device. Thismenu, such as menu 732 in FIG. 46A includes an instruction to move thewindow. It also provides a secondary menu 734 that lists the taskcurrently available in the task gallery. By moving the cursor over oneof the tasks, and releasing the secondary button of the pointing device,the user can select the destination task for the window.

After the user makes their selection, the menus disappear as shown inFIG. 46B and the virtual user is moved back in the task gallery toexpose the destination task. In FIG. 46B, the user has selected task 736as the destination task. The user interface of this embodiment thenremoves the stand associated with window 738 as shown in FIG. 46C andmoves window 738 to task 736 as shown in FIG. 46D. Window 738 is thenadded to the snapshot of task 736 as shown in FIG. 46E.

In further embodiments of the invention, the current task is replaced bythe task that received the moved window. In such embodiments, the userinterface provides an animated exchange of the two tasks as describedabove in connection with switching the current task.

Resizing Windows in the Primary Task

Under embodiments of the present invention, users may resize a window inthe primary viewing area of the current task by positioning the cursoron the edge of the window until two resizing arrows, such as resizingarrows 740 and 742 of FIG. 47A, appear. Once resizing arrows 740 and 742appear, the user depresses the primary button on the pointing device andmoves the pointing device to establish the new location for the windowborder. Such border movement is shown in FIG. 47B where border 744 hasbeen moved to the left with resizing arrows 740 and 742.

The resizing performed under the present invention differs from resizingperformed in most two-dimensional window based operating systems. Inparticular, in most two-dimensional operating systems, window resizingis performed by the application itself. However, under many embodimentsof the present invention, window resizing is performed by athree-dimensional shell, which creates the three-dimensional userinterface. In particular, the three-dimensional shell defines athree-dimensional polygon on which the image of a window is applied astexture. Thus, upon receiving a resizing instruction, thethree-dimensional shell changes the size of the polygon and reappliesthe window texturing without conveying to the application that theapplication's window has been resized. Thus, both the window and thecontents of the window are resized together under this technique of thepresent invention.

Code Block Diagram

The operation of the three-dimensional shell discussed above is morefully described in connection with the block diagram of FIG. 48 whichshows the hardware and code modules that are used in the presentinvention. In FIG. 48, an operating system 750 such as Windows® 2000from Microsoft Corporation interacts with a set of applications 752 anda three-dimensional shell 754. Applications 752 are ignorant of theexistence of three-dimensional shell 754 and are not aware that theirassociated windows are being displayed in a three-dimensionalenvironment. To accomplish this, operating system 750 andthree-dimensional shell 754 cooperate to redirect window display datafrom applications 752 into the three-dimensional environment. Theoperating system and three-dimensional shell also cooperate to modifypointing device messages before they are delivered to applications 752unless the appropriate circumstances exist in the three-dimensionalenvironment.

The method of generating a three-dimensional interface of the presentinvention by redirecting the window data generated by applications 752is discussed below with reference to the flow diagrams of FIGS. 49 and50 and the block diagram of FIG. 48. The process of FIG. 49 begins withstep 800 in which one of the applications 752 or the operating system750 determines that a window should be repainted on the display. In thiscontext, repainting the window means regenerating the image datacorresponding to the appearance of the window on the display.

After it is determined that a window needs to be repainted, theassociated application regenerates the display data at a step 802. Thisdisplay data is then sent to operating system 750. In operating systemsfrom Microsoft Corporation, the display data is routed to a graphicsdevice interface 756 (GDI.DLL) within operating system 750. Graphicsdevice interface 756 provides a standardized interface to applicationsand a specific interface to each of a collection of different types ofdisplays. Graphics device interface 756 includes a set of drawingcontexts 758 for each window generated by each of the applications 752.The drawing contexts 758 describe the location in memory where thedisplay data is to be stored so that it can be accessed by a displaydriver.

Under the present invention, instead of directing the display data to aportion of the display memory, graphics device interface 756 redirectsthe data to a location in memory denoted as redirect memory 760 of FIG.48. The redirection of the window data is shown as step 804 in FIG. 49.A further discussion of window redirection can be found in U.S. patentapplication Ser. No. 09/282,872, filed Mar. 31, 1999 and entitledDYNAMIC EFFECTS FOR COMPUTER DISPLAY WINDOWS.

After graphics device interface 756 has redirected the window displaydata, it notifies three-dimensional shell 754 that certain window datahas been updated and provides a pointer to the redirected window displaydata in redirect memory 760. This occurs at step 806 of FIG. 49. At step808, three-dimensional shell 754 marks the texture map associated withthe update window as being “dirty”.

At step 810, three-dimensional shell 754 stores a new polygon for anywindow that has had its shape changed. The polygon associated with awindow determines the location and shape of the window in thethree-dimensional display environment. For instance, in most of thescreen examples described above, each window is a texture map on arectangular polygon. By rotating and moving this polygon within thethree-dimensional environment, and then applying the associated texturemap containing the window data, the present invention can give theappearance of a three-dimensional window moving in the three-dimensionalenvironment.

The images of the task gallery and the windows in the gallery arerendered using a three-dimensional rendering toolkit 764 such asDirect3D from Microsoft Corporation. Three-dimensional rendering toolkit764 is used during an animation loop shown in FIG. 50. At step 801 ofthis loop, the location of the virtual user and the virtual user'sorientation in the task gallery is determined. The task gallery and thenon-focus tasks are then rendered at step 803 based on this userviewpoint. At step 805, three-dimensional shell 754 determines whichwindows in the focus task are in the current view. At step 810three-dimensional shell 754 determines if any of the visible windowshave had their texture map marked as dirty. If one of the visiblewindows has a “dirty” texture map, the redirected paint data is copiedinto the window's texture map at step 811. The windows are then renderedat step 812 by applying each windows texture map to its associatedpolygon.

The rendering produces display data that is stored in a back buffer 765of a display memory 766. Back buffer 765 is then swapped with a frontbuffer 767 of display memory 766 so that back buffer 765 becomes the newfront or display buffer 765. A display driver 768 then accesses newdisplay buffer 765 to generate an image on a display 770.

Three-dimensional shell 754 also receives event notification when anapplication opens a new window. Such windows include new documentwindows, dialogue boxes and drop-down menus. Three-dimensional shell 754selects a position for the new window based on the position of thewindow's parent. window and the two-dimensional location indicated forthe new window. Thus, a pull-down menu is positioned relative to itsparent window in the three-dimensional environment so that it is in thesame relative location within the parent window as it would be if bothwindows were in a two-dimensional environment. Likewise, a dialogue boxthat is designated by the application to appear in the center of thescreen is positioned relative to its parent window in thethree-dimensional environment.

Redirection of Pointer Device Inputs

In addition to redirecting the window display data created by anapplication, the present invention also modifies event data generated bya pointing device so that the event data reflects the position of thecursor in the three-dimensional environment relative to redirectedwindows that are displayed in the environment. These modifications aredescribed with reference to the flow diagram of FIG. 51 and the blockdiagram of FIG. 48.

In step 820 of FIG. 51, a pointing device driver 772 of FIG. 48generates a pointer event message based on the movement of a pointingdevice 774. Examples of pointing device 774 include a touch pad, amouse, and a track ball. Operating system 750 receives the pointer eventmessage and in step 822 determines screen coordinates for a cursor basedon the pointer event message. In operating systems from MicrosoftCorporation, the screen coordinates are determined by a dynamic linkedlibrary (DLL) shown as USER.DLL 776 in FIG. 51.

In step 824, operating system 750 notifies three-dimensional shell 754that a pointing device event has occurred. In most embodiments, thisnotification is based on an event inspection mechanism (known generallyas a low-level hit test hook) that three-dimensional shell 754 requests.With the hit test hook notification, operating system 750 includes thescreen coordinates of the cursor.

At step 832, three-dimensional shell 754 determines if the cursor isover a redirected window in the focus task that is displayed on thestage. If the cursor is not over a window in the focus task,three-dimensional shell 754 does not change the event message at step833 but instead returns the message to the operating system. Theoperating system then posts the unchanged message in the event queue forthree-dimensional shell, which uses the posted event message as inputfor changing the three-dimensional environment at step 834. For example,if the cursor is over a task located along a side wall, the floor, orthe ceiling of the task gallery, three-dimensional shell 754 may use thepointer event message as an input command for moving the task within thetask gallery. Thus, if the user clicks on the task using the pointerdevice, three-dimensional shell 754 uses the clicking input as aninstruction to make the selected task the focus task.

If the cursor is over a redirected window in the current task at step832, three-dimensional shell 754 determines the two-dimensional positionwithin the window at a step 836. Since windows within the current taskcan be rotated away from the user, the determination of thetwo-dimensional coordinates involves translating the coordinates of thecursor on the display first to a three-dimensional position in thevirtual three-dimensional environment and then to a two-dimensionalpoint on the surface of the polygon associated with the displayedwindow.

After calculating the two-dimensional position of cursor on the window,three-dimensional shell 754 determines if the window under the cursor isin the primary viewing area at step 838. If the window under the cursoris not in the primary viewing area, three-dimensional shell 754 changesthe event message by replacing the cursor's screen coordinates with thetwo-dimensional coordinates of the cursor within the window at step 840.Three-dimensional shell 754 also changes the window handle in the eventmessage so that it points at the window under the cursor and changes themessage type to a cursor over message. In other words, if the pointerevent message indicates a left button down on the pointer device,three-dimensional shell 754 would change this information into a cursorover message at step 840.

The reason for converting all pointer event messages into cursor overmessages at step 840 is that applications that are not in the primaryviewing area cannot receive pointer device input under some embodimentsof the present invention. Even so, in many embodiments of the invention,it is considered advantageous to give each application the ability tochange the shape of the cursor as the cursor moves over the applicationwindow. Thus, although an application does not receive buttoninformation when the application's window is not in the primary viewingarea, it does receive cursor over information so that it may adjust theshape of the cursor.

If the window is in the primary viewing area at step 828,three-dimensional shell 754 determines if the cursor is in the clientarea of the window at step 842. If the cursor is not in the client areaat step 842, the process continues at step 840 where the two-dimensionalwindow coordinates of the cursor are placed in the event message and awindow identifier that identifies the window below the cursor is placedin the event message.

After changing the event message at step 840, three-dimensional shell754 uses the original pointer event message information as input forchanging the three-dimensional environment at step 834. Thus, if thewindow is not in the primary viewing area, three-dimensional shell 754can use the pointer device message to move a window within the loosestack or ordered stack, or move a window between the loose stack, theordered stack and the primary view.

If the cursor is in the client area at step 842, the pointer eventmessage is changed by changing the cursor coordinates to thetwo-dimensional coordinates of the cursor over the window in thethree-dimensional environment and changing a window identifier so thatit identifies the particular window that the cursor is over. Thus, ifthe original pointer event message indicated that the left button of thepointing device had been clicked and gave the screen coordinates of thecursor during that click, three-dimensional shell 754 would replace thescreen coordinates with the two-dimensional coordinates identified bythree-dimensional shell 754. This pointer event message is then routedby operating system 750 to the application associated with theidentified window. Under this embodiment of the invention, the pointerevent message returned by three-dimensional shell 754 appears to theapplication to have come from pointing device driver 772. Thus,applications 752 are ignorant of the fact that three-dimensional shell754 exists or that their window is being displayed in athree-dimensional shell.

Although the present invention has been described with reference toparticular embodiments, workers skilled in the art will recognize thatchanges may be made in form and detail without departing from the spiritand scope of the invention. In particular, although the presentinvention has been described with reference to operating systems fromMicrosoft Corporation, the components needed will be similar on otheroperating systems. For example, a computer system that uses the X WindowSystem could be used to implement the present invention. It is notedthat for such other systems the X server should run on the same machineas the client applications and the window manager so that bitmap sharingis efficient.

1. A method of generating a display on a computer screen, the methodcomprising: displaying a non-focus task in a three-dimensionalenvironment, the non-focus task capable of including an image of atleast two windows; displaying a stage area in the three dimensionalenvironment; displaying a previous focus task in the stage area;capturing an image of the previous focus task as it appears on the stagearea; replacing the previous focus task with the image of the previousfocus task, the image of the previous focus forming a convertednon-focus task; moving the converted non-focus task away from the stagearea; moving the non-focus task to the stage area based on a user input;and making the non-focus task a focus task by displaying the at leasttwo windows such that the user can manipulate at least a portion of onewindow.
 2. The method of claim 1 wherein capturing an image of theprevious focus task comprises: moving a virtual camera from a currentposition to a preferred location in the three-dimensional environment;rendering the image of the previous focus task from the point of view ofthe virtual camera; storing the image of the previous focus task; andreturning the virtual camera to the current position.
 3. The method ofclaim 1 further comprising: displaying an image of the three-dimensionalenvironment to the user from the point of view of the current locationwhile the virtual camera is moved to the preferred location, the imageof the previous focus task is rendered and stored and the virtual camerais returned to the current position.
 4. The method of claim 1 furthercomprising: before displaying the non-focus task, displaying a menucomprising a task selection associated with the non-focus task;selecting the task selection based on input from the user; and whereindisplaying the non-focus task comprises moving a virtual camera in thethree-dimensional environment so that the non-focus task is in viewbased on the task selection.